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1.  Summary 


This  report  provides  user  and  potential  user  preferences  and  feedback  associated 
with  the  design  and  engineering  of  a  US  Army  driverless  vehicle  operating  through 
the  Warrior  Transition  Battalion  (WTB)  medical  complex  at  Fort  Bragg,  North 
Carolina.  Findings  are  derived  from  questionnaires  addressing  preferences  related 
to  perceived  trustworthiness  and  capability  of  the  autonomous  system  and  can  be 
used  to  inform  design  of  future  vehicle  attributes  and  future  research.  The 
development  and  integration  of  this  driverless  vehicle  was  part  of  the  US  Army 
Tank  Automotive  Research,  Development  and  Engineering  Center  (TARDEC) 
Applied  Robotics  for  Installations  and  Base  Operations  (ARIBO)  program.  It 
provided  door-to-door  transportation  services  between  the  WTB  medical  barracks 
and  multiple  drop-off  locations  at  Womack  Army  Medical  Center.  Passengers, 
potential  future  passengers  from  WTB,  and  bystanders  observing  the  ARIBO 
vehicle  were  given  multiple  opportunities  to  provide  feedback  during  2  phases  of 
vehicle  development:  driver-operated  (Phase  1)  and  safety-operator  enabled 
(Phase  2).  Overall  findings  were  very  positive  toward  preferences  in  the  ARIBO 
driverless  vehicle.  This  suggests  that  even  minimal  exposure  to  robotic  systems  can 
increase  positive  ratings  of  trustworthiness  and  acceptability  of  the  future 
autonomous  capabilities.  A  number  of  recommendations  were  on  design  features 
and  capabilities  to  better  improve  decision-making  and  communicate  intent  to 
passengers,  pedestrians,  and  other  road  users. 

2.  Introduction:  Applied  Robotics  for  Installations  and  Base 
Operations 

The  US  Army  TARDEC  ARIBO  program  was  designed  as  a  series  of  pilot  projects 
to  coordinate  technology  development  with  on-base  operational  needs  and 
applications  to  accelerate  the  adoption  of  autonomous  vehicle  technologies.  In 
these  pilots,  users  and  nonusers  had  the  opportunity  to  continuously  interact  with 
the  technologies  in  a  real-world,  dynamic  setting.  The  primary  advantage  of  using 
operational  military  installations  is  access  to  semi-controlled  and  well-defined, 
bounded  traffic  environments  combined  with  the  flexibility  to  modify  various 
elements  of  the  physical  infrastructure  via  the  base’s  chain  of  command  (Straub 
2015). 

Direct  feedback  on  new  technologies  can  be  a  challenge  for  the  Army.  Small 
Soldier  groups  are  typically  selected  and  asked  to  provide  feedback  on  prototype 
technologies  at  research  and  development  centers  or  carefully  controlled  field 
exercises  in  the  case  of  products  nearing  production  readiness.  The  ARIBO 

Approved  for  public  release;  distribution  is  unlimited. 

1 


program  afforded  researchers  the  opportunity  to  collect  direct  feedback  from  a 
randomized  subset  of  users  from  within  a  specific  geographic  environment.  In  this 
manner  researchers  could  evaluate  feedback  on  technologies  with  more  generalized 
relevance  while  still  addressing  warfighting-related  issues  such  as  communications, 
human-machine  interaction,  trust,  and  functional  reliability,  to  name  a  few. 

The  Fort  Bragg,  North  Carolina,  ARIBO  pilot  project  addressed  the  issue  of 
autonomous  personnel  transit.  This  project  was  designed  to  provide  on-demand 
transportation  to  wounded  Soldiers  traveling  between  the  WTB  barracks  and  the 
Womack  Army  Medical  Center  (WAMC).  Although  the  program  was  designed  for 
wounded  Soldiers  seeking  medical  treatment,  the  services  were  also  available  to 
Soldiers  and  civilians  in  the  medical  complex.  These  frequent,  short-distance 
door-to-door  trips*  between  the  medical  barracks  and  appointments  at  the  medical 
center  (Figs.  1  and  2)  provided  an  environment  to  better  understand  the  systemic 
impacts  of  driverless  vehicle  integration. 


*  The  distances  along  the  designated  routes  between  the  WTB  barracks  and  WAMC  are  between  about  1/3  and  3/4  mile. 
They  negotiate  mixed-use  pedestrian  sidewalks,  cross  a  4-lane  divided  road  around  WAMC,  and  negotiate  congested  parking 
lots  and  traffic  circles  at  the  WAMC  entrances. 
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Driverless  Robotic  Shuttle 

On  Demand  and  Online! 


The  ARIBO  Driverless  shuttle  system  is 
a  new  On  Demand  service  designed  to 
transport  you  locally.  Simply  request 
a  ride  when  and  where  you  need  it. 

Learn  more  at 

rideARIBO.com 

or  call  910-212-6832 


PICKUP/DROP-OFF  LOCATIONS: 

1  WTB  Barracks 

2  WAMC-Reilly  Rd  Entrance 

3  WAMC-Orthopedics  &  Rehab 

4  WAMC-AII  American  Enlrance 

5  Behavioral  Health 


Also  learn  how  ARIBO  is  advancing 
technology  and  providing  a  better  quality 
of  life  for  our  armed  forces.*  Find  out 
bow  you  can  be  part  of  getting  robotics 
into  the  hands  of  our  vrarfighters 


iair  Acci 


Fig.  1  ARIBO  advertisement  at  Fort  Bragg  identifying  vehicle  pickup  and  drop-off 
locations 
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Fig.  2  Planned  vehicle  routes  between  WTB  and  WAMC 

Research  efforts  included  comparative  analysis  between  human  decisions  and 
autonomy,  operational  efficiency  impact,  user  and  nonuser  behavior,  trust,  and 
acceptance,  as  well  as  vehicle  platform  and  technology  development,  engineering, 
maintenance,  and  reliability  assessments.  Functionally,  the  service  provided  a 
convenient  mode  of  transportation  and  an  opportunity  to  increase  on-time 
appointment  rates  for  Soldiers/ 

A  phased  introduction  of  autonomous  control  was  implemented  to  ensure  safe 
operations  and  a  smooth  increase  in  trust  and  confidence  of  the  system.  Phase  1 
consisted  of  a  human  driver-operated  shuttle  service.  During  this  phase,  vehicle 
data  were  collected  for  testing  the  underlying  architecture  and  autonomy  algorithms 
while  a  human  driver  controlled  the  vehicle.  The  purpose  of  this  phase  of 
development  was  2-fold:  first,  because  the  vehicle  was  operated  by  a  trained  human 


*  Missed  appointments  account  annually  for  about  $1.2  million  in  opportunity  cost. 
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driver,  we  were  able  to  evaluate  the  performance  of  the  sensing  and  control  systems 
in  real-world  conditions  without  increasing  risk  to  any  passengers;  and  second,  we 
were  able  to  evaluate  the  autonomy’s  performance  not  only  against  real-world 
conditions,  but  compare  performance  and  decision-making  against  the 
human-driven  baseline  using  a  recording  scheme  and  automated  analysis  based  on 
researcher-defined  upper  and  lower  limits  of  acceptable  performance.  Results  of 
the  analysis  provided  data  for  the  decision  to  proceed  to  the  next  phase  of  the 
project.  Phase  2  transitioned  the  role  of  the  driver  to  that  of  a  safety  operator.  A 
safety  operator  is  trained  in  manual  operations  as  well  as  the  ability  to  transition 
between  autonomous  control  mode  and  manual  control  mode.  The  safety  operator 
sits  in  the  driver’s  seat,  monitoring  the  status  of  the  vehicle  and  the  environment. 
The  role  of  the  safety  operator  was  to  only  intervene  with  the  vehicle’s  autonomy 
in  case  of  an  emergency  or  potential  vehicle  error. 

2.1  ARIBO  Vehicle 

The  vehicles  stationed  at  Fort  Bragg  were  Cushman  Shuttle  6  low-speed  electric 
vehicles  (Fig.  3).  These  vehicles  were  capable  of  being  optionally  manned,  meaning 
they  could  run  either  in  fully  autonomous,  driverless  mode  or  in  manual, 
human-operated  mode.  Operations  in  human  mode  were  no  different  from  the 
normal  operations  of  a  Cushman  Shuttle  6  one  might  encounter  in  any  similar 
environment. 


Fig.  3  ARIBO  driverless  vehicle  variants:  a)  6-passenger  and  b)  Americans  with 
Disabilities  Act  (ADA)-compliant  wheelchair  accessible  models 

The  autonomy  package  included  a  number  of  sensors  to  maintain  situation 
awareness  of  the  environment  and  operate  within  the  dynamic  nature  of  the  medical 
complex  (Fig.  4).  Two  Velodyne  VLP-16  lidars  mounted  on  the  passenger-side 
front  and  driver-side  rear  provided  a  360°  horizontal  and  vertical  field  of  view  with 
16  line  scans.  These  are  the  primary  sensors  used  to  detect  objects  around  the 
vehicle  and  localize  against  curbs  and  other  known  landmarks.  Two  Hokuyo 
single-line  lidar  sensors  were  mounted  to  the  front  of  the  vehicle,  one  pointing 
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forward  and  one  pointing  down.  The  forward-facing  Hokuyo  served  as  a  redundant 
obstacle  detection  sensor.  The  downward-facing  sensor  was  used  for  identifying 
cross-track  localization  error  by  detecting  lane  markings  and  curbs.  Two 
differential  GPS  antennas  (one  on  the  front  center  and  the  other  on  the  rear  center 
of  the  roof)  provided  redundant  heading  measurements  and  precision  localization 
for  the  vehicle.  Data  radio  antennas  are  used  to  communicate  to  the  control  station 
at  the  duty  desk  and  with  the  reservation  system.  Automotive-grade  radar  sensors 
were  added  to  the  front  sides  of  the  vehicle  to  provide  highly  reliable  detection  of 
vehicle  cross  traffic  at  intersections. 


Fig.  4  Vehicle  sensors 


Cameras  were  positioned  in  the  cab  of  the  vehicle  facing  forward  (Fig.  5a)  and 
toward  the  passengers  (Fig.  5b).  The  cameras  were  for  passenger  safety  and  to 
record  the  environment.  Video  data  were  recorded  to  Secure  Digital  cards  on  the 
cameras  and  to  an  on-board  computer.  The  video  was  downloaded  daily  to  a 
secured,  encrypted  computer  accessible  only  to  the  researchers. 


Fig.  5  a)  Forward-facing  and  b)  passenger-facing  cameras 
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Five  emergency  stop  (e-stop)  buttons  were  available  that  disengage  the  autonomy 
system,  bring  the  vehicle  to  a  full  stop,  and  engage  the  parking  brake  (Fig.  6).  If 
activated  at  top  speed  (15  mph),  the  e-stop  halted  the  vehicle  in  about  5.5  m.  While 
in  human-mode,  the  e-stop  system  was  disengaged  to  eliminate  the  risk  of  it  being 
engaged  while  a  person  is  driving  the  shuttle. 


Fig.  6  Placement  of  e-stop  buttons 


2.2  Current  Work 

The  research  objective  of  the  work  described  in  this  report  was  to  collect  user 
perception  data  associated  with  the  perceived  functionality  and  trustworthiness  of 
the  driverless  vehicle  during  the  first  2  phases  of  operation.  Questionnaire  data  were 
anonymously  collected  from  individuals  (Soldiers  and  civilians)  who  may  have 
been  a  passenger  on  or  observed  the  ARIBO  vehicles  operating  throughout  the 
WTB  medical  complex.  The  purpose  of  this  study  was  to  expand  on  results  from 
previous  simulation  experiments  involving  trust  with  the  ARIBO  driverless 
vehicles  to  more  directly  inform  future  autonomous  vehicle  and  user  interface 
designs  required  for  effective  human-robot  teaming. 

3.  Background:  Development  of  the  Questionnaire 

The  questionnaire  items  were  built  on  previous  research  efforts  to  understand  the 
relationship  between  trust,  use,  and  robot  design.  As  trust  is  essential  for  the 
successful  function  of  any  team  (Groom  and  Nass  2007),  the  use  of  a  robot  is 
directly  related  to  the  human’s  ability  to  place  trust  in  that  robot  (Lussier  et  al. 
2007).  Hancock  et  al.  (2011)  identified  through  meta-analytic  procedures  that  the 
primary  moderator  of  human-robot  trust  development  is  the  perception  of  the 
functional  capabilities  of  the  robot.  However,  the  perceptions  of  the  functional 
capabilities  of  the  robot  are  not  exhibited  through  actual  behaviors  alone.  Duffy 
(2003)  suggested  the  importance  of  the  physical  form  of  the  robot,  since  it  is  the 
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compatibility  of  the  design  with  the  functional  capability  that  increases  acceptance 
of  the  robot.  Further,  Schaefer  and  colleagues  (2012)  found  that  individuals  begin 
to  make  perceptions  about  the  potential  functionality  of  the  robot  (e.g.,  perceived 
level  of  automation  and  perceived  robot  intelligence),  future  outcomes  with  the 
robot  (e.g.,  future  use),  and  trustworthiness  based  on  physical  form,  even  depicted 
through  a  static  image.  Perceived  intelligence,  perceived  level  of  automation,  and 
intended  future  use  were  also  highly  correlated  with  trustworthiness  ratings 
(Schaefer  2013).  Therefore  items  specific  to  vehicle  intelligence,  control,  use, 
safety,  and  trustworthiness  as  well  as  perceptions  regarding  specific  driverless 
vehicle  behaviors  (e.g.,  navigation,  obstacle  detection  and  avoidance,  and  ability  to 
follow  road  rules)  were  included  in  the  development  of  the  questionnaire. 

This  questionnaire  was  also  designed  to  provide  recommendations  specific  to  the 
design  of  this  driverless  vehicle  that  could  potentially  increase  user  comfort  or 
engender  trust  in  the  vehicle.  These  items  were  based  around  previous  research 
efforts  and  developed  to  benefit  future  research  efforts  specific  to  passengers  and 
bystanders  (Straub  and  Schaefer  2015).  In  an  earlier  simulation-based  study,  the 
respondents  responded  to  open-ended  questions  specific  to  the  vehicle  design 
recommendations.  These  resulted  in  4  main  areas  of  missing  feedback  from  the 
vehicle:  navigation-related  feedback,  proximity-related  information,  system  state, 
and  passenger  safety  (see  Appendix  A  for  further  details).  The  primary  sentiment 
of  this  prior  research  was  that  better  shared  situation  awareness  (i.e.,  the  capability 
that  the  vehicle  is  aware  of  the  physical  environment  and  intends  to  respond 
appropriately)  will  engender  trust  and  increase  comfort  in  the  vehicle  (see  Schaefer 
and  Straub  2016). 

4.  Methodology 

Data  were  collected  using  a  questionnaire.  The  questionnaire  was  available  to  both 
passengers  and  bystanders,  also  referred  to  as  observers.  A  passenger  was  someone 
who  received  a  ride  from  one  point  to  another  with  no  responsibility  for  the  safety 
of  others  or  the  safe  operation  of  the  vehicle,  whereas  a  bystander  was  someone 
who  simply  witnessed  the  vehicle  in  operational  mode.  Respondents  included  both 
Soldiers  and  civilians  who  previously  scheduled  a  ride  or  those  who  were  recruited 
by  email  from  the  battalion  command.  Respondents  with  scheduled  rides 
automatically  received  an  electronic  version  of  the  questionnaire  following  their 
ride.  Others  had  the  opportunity  to  complete  either  the  electronic  version  via  a  web 
link  on  their  own  time  or  the  paper  version  in  person  immediately  following  the 
ride. 
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4.1  Questionnaire 


A  16-item  author-created  questionnaire  was  accessible  via  paper  and  pencil,  as  well 
as  through  SurveyMonkey.com,  an  online  questionnaire  software  program  that  can 
be  accessed  from  computer  or  smart  phone.  Appendix  B  includes  2  versions  of  the 
questionnaire  for  each  phase  of  operation.  Phase  1  items  were  worded  in  the  future 
tense  since  a  driver  was  operating  the  vehicle,  while  Phase  2  items  were  worded  in 
the  present  tense  since  the  vehicle  was  operating  autonomously. 

The  first  4  items  were  designed  to  gather  information  about  the  respondent’s 
interaction  with  the  vehicle,  number  of  times  as  a  passenger,  first  time  filling  out 
questionnaire,  and  if  they  had  seen  the  vehicle  operate  autonomously  (even  if  they 
never  rode  on  the  vehicle).  Items  5-9  were  7-point  Likert-type  scale  items  related 
to  questions  used  in  previous  work  to  identify  perceptions  about  the  trustworthiness 
of  specific  robots  (Schaefer  et  al.  2012;  Schaefer  2013).  Items  10-13  were  specific 
to  the  acceptability  of  the  autonomy-enabled  capabilities  of  the  vehicle.  The  final 
open-ended  questions  were  included  to  help  direct  future  research.  They  included 
items  that  built  on  previous  simulation  research  to  identify  specific  design  elements 
that  could  be  used  to  increase  comfort  or  engender  trust  while  riding  in  the  vehicle 
(Schaefer  and  Straub  2016),  and  increase  comfort  or  engender  trust  for  pedestrians 
(Straub  and  Schaefer  2015).  No  personally  identifiable  information  was  collected 
during  this  study. 

4.2  SurveyMonkey.com 

Following  all  scheduled  rides,  respondents  received  an  automatically  generated 
email  or  were  handed  a  flyer  with  the  opportunity  to  complete  the  questionnaire  via 
SurveyMonkey.com.  SurveyMonkey.com  was  selected  because  it  has  been 
approved  for  use  by  the  US  Army  Research  Laboratory’s  Information  Assurance 
Management  Office.  Backup  paper  and  pencil  versions  of  the  questionnaire  were 
also  available.  If  a  paper  and  pencil  version  was  completed,  then  data  were  entered 
into  the  SurveyMonkey.com  web  link  so  that  all  of  the  data  were  complied. 

4.3  Respondents 

Respondents  consisted  of  Soldiers  and  civilian  personnel  who  1)  had  scheduled 
transportation  using  the  ARIBO  driverless  vehicle,  2)  had  observed  the  vehicle 
operating  throughout  the  WTB  medical  complex,  or  3)  were  members  of  the  WTB 
that  may  or  may  not  have  seen  the  vehicle  operating  autonomously  throughout  the 
medical  complex.  A  total  of  26  responses  were  recorded.  These  included  7 
responses  during  Phase  1  (July-Oct  2016)  during  which  the  vehicle  was  driven  by 
a  human  operator  and  19  responses  during  Phase  2  (Nov  2016-Aug  2017)  when 
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the  human  driver  relinquished  driving  control  to  the  vehicle  and  assumed  the  role 
safety  operator. 

5.  Results  and  Discussion 


The  user  preference  questionnaires  were  completed  by  the  driver  or  safety  operator 
of  the  vehicle  (N  =  3),  passengers  (N  =  16),  bystanders  or  observers  (N  =  6),  and  1 
Soldier  in  the  WTB  who  did  not  report  being  a  passenger  or  bystander.  More 
specifically,  out  of  the  26  responses,  7  people  never  rode  the  vehicle;  10  people  had 
between  1  and  4  rides  on  the  vehicle;  3  people  rode  the  vehicle  between  5  and  10 
times;  and  6  people  rode  the  vehicle  more  than  10  times  (see  Fig.  7).  All  26 
respondents  saw  the  vehicle  operating  in  the  area  surrounding  the  WTB  Medical 
Complex,  but  only  15  reported  seeing  the  vehicle  operate  autonomously  (2  from 
Phase  1  and  13  from  Phase  2).  The  remainder  of  this  section  describes  perceptions 
of  trustworthiness,  perceptions  of  autonomous  vehicle  capabilities,  and 
recommendations  for  design  guidelines  to  increase  comfort  and  trust  of  both  riders 
and  pedestrians. 


■  Phase  1  Responses  □  Phase  2  Responses 


Rides 

Number  of  Rides  on  the  Vehicle 


Fig.  7  Self-reported  ridership  numbers  for  Phase  1  and  Phase  2  of  vehicle  operation 

5.1  Trustworthiness 

The  questions  to  assess  user  perceptions  of  trustworthiness  included  ratings  of 
perceived  levels  of  vehicle  intelligence,  autonomy  (i.e.,  the  vehicle’s  ability  to 
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control  itself),  trustworthiness,  level  of  use,  and  safety.  Twenty-four  out  of  26 
respondents  answered  these  questions.  Overall,  results  demonstrated  positive 
agreement  in  support  of  perceived  intelligence,  perceived  level  of  autonomy,  and 
perceived  trustworthiness  of  the  ARIBO  vehicle  (Fig.  8). 


Perceived  Intelligence 
7.69%  0-°°O/“3.85% 


88.46% 


■  Negative  Neutral  Positive  No  response 


Perceived  Autonomy 

7.69%  3.85%3  85„/o 


84.62% 


■  Negative  Neutral  ■  Positive  No  response 


Perceived  Trustworthiness 
11.54%  0.00%  7.69% 


80.77% 

■Negative  Neutral  Positive  No  response 


Perceived  Safety  Perceived  Use 

7.69%  3.85%  7.69%  11 


■  Negative  Neutral  Positive  No  response 


■  Negative  Neutral  Positive  No  response 


Fig.  8  Perceived  trustworthiness  ratings 
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5.1.1  Perceived  Intelligence  and  Autonomy 

The  first  2  items  looked  at  perceived  vehicle  intelligence  and  perceived  level  of 
autonomy.  Overall,  both  items  were  perceived  positively  by  both  passengers  and 
bystanders.  The  percentage  of  participants  that  rated  positive  agreement  of  the 
perceived  intelligence  of  the  vehicle  was  88.46%  (3  somewhat  agree,  5  agree,  and 
15  strongly  agree),  while  84.62%  of  participants  rated  the  perceived  autonomy  item 
positively  (2  somewhat  agree,  7  agree,  and  13  strongly  agree).  Only  one  participant 
rated  both  items  as  neutral.  This  participant  never  rode  or  observed  the  vehicle 
operating  autonomously.  Later  open-ended  questions  suggested  that  this  participant 
also  desired  more  feedback  from  the  vehicle  to  quantify  the  vehicle’s  decision¬ 
making  processes  and  autonomous  capabilities.  These  findings  suggest  that 
individuals  want  to  believe  that  the  vehicle  will  be  intelligent  and  have  the 
capability  to  function  without  the  direct  control  of  an  operator,  even  ones  that  have 
never  seen  the  vehicle  operate  autonomously  as  seen  in  responses  from  Phase  1 
operations.  Results  from  Phase  2  operations  suggest  that  individuals  make 
determinations  about  the  level  of  vehicle  intelligence  and  autonomy  even  though 
the  vehicle  was  not  designed  to  have  any  directed  feedback  as  to  the  specific 
intelligence  architecture.  This  is  important  to  the  Army  because  it  suggests 
perceptions  about  the  driverless  vehicle’s  ability  are  based  on  relatively  few 
performance  variables.  Additionally,  the  Army  spends  more  every  year  to  run  its 
installations  than  just  about  anything  other  than  personnel  costs.  User  and  potential 
user  perceptions  are  important  to  the  acceptance  of  driverless  vehicles.  Automation 
can  improve  efficiencies  and  safe  operations  on  bases  and  the  degree  to  which  it  is 
widely  adopted  can  have  a  positive  impact  on  budgets  and  operational  performance. 
Diving  further  into  the  factors  that  lead  to  these  few  perceptions  of  ability  or 
competency  of  such  vehicles  can  help  developers  focus  on  those  elements  most 
important  to  Soldiers  in  the  field. 

5.1.2  Perceived  Trustworthiness 

Self-report  ratings  of  vehicle  trustworthiness  were  positively  rated  such  that 
80.77%  of  respondents  rated  the  perceived  trustworthiness  positively  (2  somewhat 
agree,  8  agree,  and  11  strongly  agree).  There  were  no  negative  ratings,  and  only  2 
neutral  ratings.  Out  of  the  2  neutral  ratings,  one  respondent  had  never  ridden  or 
observed  the  vehicle  operating  autonomously,  and  the  second  had  previously  ridden 
the  vehicle  5-10  times  during  Phase  1  (operated  by  the  human  driver),  but  had  seen 
it  operate  autonomously  as  an  observer. 
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These  overall  positive  findings  could  directly  support  the  current  design  and 
functional  capabilities  of  the  vehicle  for  short  duration  travel,  such  that  the  current 
design  features  and  capabilities  provided  adequate  situation  awareness  needed  to 
calibrate  trust  in  the  vehicle  with  minimal  prior  interaction.  However,  these  positive 
trustworthiness  ratings  may  also  be  influenced  by  the  larger  system  including  the 
human  driver  or  safety  operator  onboard  the  vehicle,  as  well  as  command  support 
and  encouragement  of  ridership  with  the  ARIBO  driverless  vehicle  at  WAMC.  In 
addition,  the  2  neutral  responses  suggest  some  individuals  may  need  to  physically 
see,  experience,  and  understand  the  capabilities  of  the  system  before  putting  blind 
faith  in  the  system. 

5.1.3  Perceived  Use 

The  perceived  use  question  asked  respondents  if  they  felt  the  vehicle  would  be  used 
regularly  by  the  Soldiers  at  WTB,  not  just  when  operated  by  a  driver  or  safety 
operator  but  also  when  the  operator  was  removed  from  the  vehicle.  While  there  was 
slightly  more  divergence  in  responses,  just  under  70%  of  respondents  rated  the 
perceived  use  positively  (6  somewhat  agree,  5  agree,  7  strongly  agree).  These  data 
suggest  that  to  some  degree  Soldiers  and  civilians  believe  their  fellows  will  value 
point-to-point,  scheduled,  and  on-demand  driverless  transportation.  However,  there 
was  some  divergence  in  perceived  amount  of  vehicle  use:  2  strongly  disagreed  with 
this  item,  1  disagreed,  and  3  were  neutral. 

Respondents  with  these  negative  ratings  for  perceived  use  had  limited  to  no  prior 
interaction  with  the  vehicle  during  Phase  1  or  Phase  2  of  operations  and  had  some 
limited  knowledge  or  understanding  of  how  the  vehicle  could  or  would  operate 
safely  without  a  driver.  This  supports  the  need  for  increased  access  in  the  Army  to 
autonomous  vehicle  technologies  to  increase  understanding  and  trust  to  better 
support  appropriate  use  of  future  robotic  systems. 

The  3  neutral  ratings  came  from  2  people  on  their  first  ride  and  1  who  had  ridden 
the  vehicle  more  than  10  times.  While  some  undirected  feedback  supported  that 
these  respondents  would  be  willing  to  use  the  vehicle  again,  there  was  some 
hesitation  on  rating  how  others  would  use  the  vehicle.  This  was  in  part  due  to  the 
limited  knowledge  of  the  larger  population  base  regarding  vehicle  availability, 
limited  hours  of  operation,  and  observations  of  vehicle  runs  without  passengers.  In 
addition,  these  data  do  not  control  for  the  presence  of  the  safety  operator,  making 
it  difficult  to  determine  if  respondents  would  recommend  use  of  the  vehicle  with 
the  safety  operator  behind  the  wheel  although  our  data  (Section  5.1.1)  on 
trustworthiness  suggests  they  would.  Once  the  technology  reliably  and  safely 
allows,  future  research  should  examine  these  questions  without  the  presence  of  a 
safety  operator. 
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5.1.4  Perceived  Safety 

Ratings  for  perceived  safety  were  slightly  higher  with  76.92%  rating  perceived 
safety  positively  (3  somewhat  agree,  6  agree,  and  11  strongly  agree).  While  it  is 
possible  that  these  ratings  support  the  design  and  capabilities  of  the  system  to 
promote  safety,  it  is  also  possible  that  some  of  these  ratings  were  not  based  on 
interaction  or  observation  of  the  vehicle  but  rather  from  the  larger  assumption  that 
the  Army  would  not  allow  vehicles  on  the  road  that  were  not  safe.  This  supports 
the  idea  that  trust  in  autonomy  has  larger  implications  than  just  improving 
engineering  systems  alone.  This  type  of  trust  is  related  to  the  vehicle,  but  also  the 
larger  Army  and  support  of  the  command.  Future  work  would  need  to  be  conducted 
to  address  this  more  fully. 

These  4  respondents  had  either  neutral  safety  ratings  (N  =  3)  or  felt  the  vehicle  was 
unsafe  (N  =  1).  A11  these  respondents  had  limited  interactions  with  the  autonomy 
features  and  capabilities  of  this  vehicle.  Therefore,  it  is  possible  to  speculate  that 
these  evaluations  may  have  been  based  on  some  bias  or  preconception  of  the 
technology,  or  influenced  by  the  safety  performance  of  the  human  driver.  While 
additional  research  is  needed  to  determine  the  specific  reasoning  behind  these 
ratings,  this  finding  supports  the  Army’s  need  for  increased  access  and  exposure  to 
robotic  technologies.  This  is  important  to  advancing  shared  situation  awareness  and 
understanding  of  future  robotic  capabilities  to  appropriately  calibrate  trust  and 
promote  effective  use. 

5.2  Acceptability  of  Vehicle  Autonomy 

Over  the  course  of  3  years,  the  ARIBO  vehicle  has  demonstrated  the  capability  to 
effectively  navigate  through  the  medical  complex;  detect  and  avoid  obstacles,  other 
road  users,  and  pedestrians;  and  follow  traffic  rules  while  operating  in  fully 
functioning  driverless  operations  (with  a  safety  operator  present).  However,  actual 
system  capabilities  may  not  directly  match  perceived  capabilities  from  passengers 
or  bystanders  with  limited  knowledge  of  the  functional  capabilities  of  the  vehicle. 
Therefore,  respondents  assessed  the  perceived  acceptability  of  the  vehicle’s 
autonomous  navigation,  obstacle  detection  and  obstacle  avoidance  (ODOA),  and 
the  ability  to  follow  traffic  rules.  While  respondents  may  have  never  seen  or 
personally  interacted  with  the  vehicle  during  autonomous  operations,  the  results 
provide  some  insight  into  the  user  population’s  perceptions  of  future  autonomy. 

Respondents  rated  3  capabilities:  autonomous  navigation,  ODOA,  and  following 
road  rules.  Overall,  80.77%  of  the  respondents  found  the  autonomous  vehicle 
navigation  capabilities  to  be  acceptable,  while  73.08%  found  the  ODOA  and  ability 
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to  follow  road  rules  to  be  acceptable.  Furthermore,  there  were  no  significant 
differences  between  respondents  who  had  little  to  no  interaction  with  the  ARIBO 
vehicle  and  those  who  were  passengers  during  driverless  operations.  This  suggests 
that  overall,  individuals  support  the  current  and  future  capabilities  of  the  vehicle. 

However,  there  is  still  room  to  advance  the  engineering  and  design  needed  for 
increased  understanding  and  support  of  these  vehicle  capabilities.  The  additional 
negative  and  neutral  ratings  on  these  3  items  provide  some  insight  into  the  types  of 
individuals  and  amount  of  information  access  that  may  be  needed  to  increase 
acceptability  ratings  of  these  types  of  vehicle  autonomy.  For  example,  3  individuals 
had  neutral  (N  =  1)  or  slightly  unacceptable  (N  =  2)  ratings  regarding  the 
navigation-based  autonomy.  These  respondents  either  had  limited  prior  interaction 
with  the  vehicle  or  had  concerns  about  the  safety-based  navigation  and 
recommended  more  direct  vehicle  feedback  for  passengers.  In  addition,  the 
acceptability  ratings  of  the  ODOA-based  autonomy  included  3  neutral  ratings,  a 
slightly  unacceptable  rating,  and  an  unacceptable  rating.  Similarly,  all  these 
respondents  had  limited  prior  interactions  with  the  autonomous  capabilities  of  the 
vehicle.  This  finding  suggests  that  perceived  capabilities  will  be  enhanced  either 
through  increased  access  to  robotic  systems  through  communication  of  capabilities 
or  actual  interaction.  The  acceptability  of  the  vehicle  to  follow  traffic  rules  ratings 
included  3  neutral  and  1  slightly  unacceptable  rating.  Most  of  these  respondents 
had  limited  or  no  prior  interaction  with  the  vehicle,  suggesting  that  similar  to  the 
above  finding,  increased  communication  of  capabilities  or  interaction  with  the 
system  can  increase  understanding  of  the  capabilities.  Further,  a  respondent  was  a 
prior  passenger  on  the  vehicle  but  had  concerns  of  the  safety  of  the  vehicle 
following  concerns  related  to  right-of-way  behaviors  throughout  an  intersection. 
This  suggests  that  there  may  be  a  benefit  to  share  more  information  or  feedback 
during  an  interaction  to  better  communicate  vehicle  intent. 

Overall,  the  data  above  indicate  confidence  in  the  vehicle’s  autonomy,  navigation, 
and  ability  to  detect  and  avoid  obstacles.  This  suggests  that  autonomous  vehicle 
performance  capable  of  approximating  human  performance  is  what  passengers  and 
bystanders  expect  of  autonomous  vehicles.  Further  findings  suggest  that  there  may 
be  a  subset  of  people  who  have  limited  exposure  to  autonomy  who  would  benefit 
from  increased  exposure  and  feedback  of  autonomous  capabilities  for  navigation, 
ODOA,  and  the  capability  to  follow  road  rules. 
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5.3  Recommendations  for  Vehicle  Design 


All  respondents  were  given  the  opportunity  to  provide  design  feedback  and 
recommendations  related  to  rider  comfort,  pedestrian  comfort,  rider  trust,  and  a 
future  user  display  design. 


5.3.1  Rider  Comfort 

All  respondents  were  asked  to  provide  open-ended  recommendations  for  the  future 
design  of  the  vehicle  to  increase  comfort  of  passengers  onboard  the  vehicle.  Out  of 
the  26  respondents,  only  6  provided  directed  comments  for  enhancing  passenger 
comfort.  The  remaining  20  respondents  left  this  item  blank,  specifically  denoted 
that  they  had  no  direct  feedback,  or  stated  the  vehicle  provided  adequate  passenger 
comfort  for  the  short  duration  of  the  trip.  Table  1  lists  the  specific  feedback  and 
comments. 


Table  1  Passenger  comfort  feedback  and  vehicle  design  guidelines 


Number  of 
comments 

2 

1 

1 


Specific  feedback 

Second  row  has  limited  leg  room  for  tall  passengers  or  those  with  severe  injuries 
Seats  are  uncomfortable,  especially  for  those  with  injuries 
Need  seat  belts3 


9  Need  feedback  from  vehicle  or  signage  to  communicate  vehicle  capabilities  when 

interacting  with  other  road  users  in  intersections,  parking  lots,  and  on  roadway 
“All  seats  were  instrumented  with  seatbelts.  Recommendation  came  from  person  who  never  rode  in 
the  vehicle. 


Two  major  types  of  feedback  included  design  recommendations  for  physical 
comfort  and  for  providing  passenger  safety.  The  first  was  directed  at 
accommodations  for  this  specific  use  population,  including  more  comfortable  seats 
and  extended  leg  room.  This  accommodation  was  considered  in  the  mobile 
application  when  scheduling  a  ride;  a  wheelchair-accessible  vehicle  with  extended 
leg  room  was  available  for  transport  and  could  be  selected  at  the  time  of  reservation. 
The  second  design  recommendation  was  specific  to  both  physical  safety  and 
perceived  safety.  Seatbelts  were  already  included  for  every  seat  on  the  vehicles.  In 
addition,  the  need  for  communication  to  alleviate  discomfort  and  mitigate 
uncertainty  is  paramount  for  increasing  passenger  comfort  in  the  vehicle,  especially 
when  the  safety  operator  is  removed  from  the  driver’s  seat.  Future  vehicle  design 
considerations  and  research  for  driverless  shuttles  should  address  mechanisms  to 
effectively  communicate  safety  options  available  to  passengers.  Passenger  comfort 
and  safety  features  are  clearly  important  to  passengers  and  potential  passengers  of 
driverless  transport  vehicles.  Features  and  tasks  requiring  action  on  the  part  of 
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passengers  (e.g.,  fasten  seat  belts)  should  be  clear  from  early  in  the  reservation 
process  to  embarkation  through  transport  and  debarkation. 


5.3.2  Pedestrian  Comfort 

Since  these  vehicles  operate  on  both  roadways  and  sidewalks,  all  respondents  were 
asked  to  provide  open-ended  recommendations  for  the  future  design  of  the  vehicle 
to  increase  comfort  of  pedestrians  near  the  vehicle.  Out  of  the  26  respondents,  8 
respondents  provided  feedback  related  to  pedestrian  comfort.  The  remaining  18 
respondents  left  this  item  blank,  specifically  denoted  that  they  had  no  direct 
feedback,  or  felt  the  vehicle  had  no  direct  effect  on  pedestrians.  Table  2  provides 
specific  feedback  and  comments. 


Table  2  Design  guidelines  for  pedestrian  comfort  and  safety 


Number  of 
comments 


Specific  feedback 


4  Addition  of  audible  alerts  or  warnings 

1  Use  of  a  flashing  light  so  others  can  see  the  AIRBO  coming11 

1  Larger  vehicle  to  increase  pedestrian  awareness 

I  Inclusion  of  a  large,  legible  sign  onboard  the  vehicle  stating  that  vehicle  operates 

safely  without  a  driver 

1  Some  type  of  warning  is  needed  when  approaching  pedestrians 

a  Both  vehicles  were  instrumented  with  a  yellow  safety  light. 


All  design  recommendations  were  specific  to  providing  additional  pedestrian 
safety.  Specifically,  the  respondents  were  interested  in  providing  pedestrians  with 
awareness  of  the  vehicle’s  presence  and  capabilities  through  means  such  as  signage, 
or  audible  alerts  or  warnings.  Some  examples  included  a  verbal  acknowledgement 
of  when  the  vehicle  was  paused  for  a  pedestrian  or  a  hom  beep  to  let  a  pedestrian 
know  the  vehicle  was  waiting  on  them.  Future  research  and  vehicle  design  should 
examine  common  communication  techniques  used  to  communicate  with 
able-bodied  adult  pedestrians,  children,  and  pedestrians  with  disabilities  (e.g.,  blind 
or  deaf),  as  well  as  the  implication  of  regional  norms  or  variations  in  order  to  assess 
different  types  of  alerts  and  warnings  that  could  be  used  to  effectively  communicate 
vehicle  intent  with  pedestrians. 


5.3.3  Passenger  Trust 

All  respondents  were  asked  to  provide  recommendations  for  the  design  of  the 
vehicle  that  would  increase  future  rider  trust  in  the  vehicle,  especially  once  the 
safety  operator  was  removed  from  the  driver’s  seat.  Out  of  the  26  respondents,  5 
provided  design  guidance  on  improving  rider  trust  in  the  driverless  vehicle.  The 
remaining  21  respondents  left  the  item  blank,  directly  stated  no  guidelines,  or 
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specifically  pointed  out  that  current  design  did  not  lessen  trust  in  the  vehicle.  All  5 
of  the  comments  were  related  to  passenger  awareness  and  ways  to  improve  the 
passengers’  perception  of  safety  in  terms  of  error  prevention  and  mitigation,  such 
as  means  for  adapting  control  allocation.  Table  3  provides  specific  feedback  and 
comments. 


Table  3  Design  guidelines  to  increase  passenger  trust 


Number  of 
comments 


Specific  feedback 


I  Change  outward  vehicle  appearance  (e.g.,  bright  color)  to  increase  nonuser 

awareness 

1  Include  eye-catching  signage  of  features  onboard  the  vehicle 

1  A  communication  system  with  a  human  for  emergencies 

1  Include  an  emergency  stop  button  that  the  passengers  could  press3 

1  Capability  to  take  control  of  the  vehicle  if  unsafe 

a  Both  vehicles  were  instrumented  with  multiple  emergency  stop  buttons  (see  Fig.  6). 


Development  of  appropriate  vehicle  autonomy  and  trust  requires  an  understanding 
of  how  driverless  vehicles  and  other  road  users  will  mutually  interact  during  shared 
operations  (Straub  and  Schaefer  in  press).  The  technology  is  not  yet  available  for 
the  general  population  to  be  familiar  with  autonomous  vehicles.  Access  to  these 
types  of  technologies  will  increase  understanding  and  experience  with  the 
technology,  which  may  lead  to  an  increase  in  trust.  It  is  difficult  for  people  to  trust 
something  they  do  not  understand  or  do  not  know  about.  Results  from  this  open- 
ended  feedback  suggested  that  design  guidelines  should  include  both  indirect  and 
direct  error  prevention  and  mitigation  strategies. 


Specific  form-based  features  can  support  (or  hinder)  expectations  of  vehicle 
capabilities  (Groom  and  Nass  2007).  For  example,  a  feature  such  as  changing  the 
outward  appearance  of  the  vehicle  to  increase  other  road-user  awareness  indirectly 
mitigates  issues  that  could  occur  when  sharing  the  roadway,  thus  reducing  the 
possibility  of  misunderstanding  vehicle  intent.  Similarly,  onboard  signage  of 
vehicle  features  or  a  communication  system  with  person  monitoring  the  system  can 
increase  the  passenger’s  situation  awareness  of  what  the  vehicle  can  and  should  be 
doing  in  certain  situations. 


Moreover,  once  the  safety  operator  is  removed  from  the  driver’s  seat,  there  will  be 
questions  as  to  how  passengers  will  be  able  to  interact  with  vehicle  and  concern  for 
how  to  respond  in  case  of  errors.  At  present,  passengers  have  limited  familiarity 
with  the  ARIBO  vehicle  and  there  is  limited  feedback  onboard  to  communicate  its 
state,  intended  behaviors,  or  awareness  of  the  environment  including  pedestrian  or 
vehicle  awareness.  This  could  directly  impact  passenger’s  comfort  and  trust  and 
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directly  impact  the  need  to  take  control  of  the  vehicle  in  what  could  be  perceived 
to  be  a  harmful  situation.  An  example  of  a  harmful  situation  was  when  the 
autonomous  vehicle  was  in  an  intersection  and  another  vehicle  was  approaching, 
there  was  no  indicating  that  the  other  vehicle  was  sensed.  Therefore,  specific  design 
recommendations  for  direct  error  mitigation  included  the  capabilities  to  take  over 
control  either  through  use  of  emergency  stop  buttons  or  full  manual  controls.  This 
leads  to  a  larger  issue  of  not  just  having  these  design  features  available,  but 
appropriate  use  of  these  features  and  responsibility.  Appropriate  control  allocation 
can  only  be  established  when  the  riders  and  vehicle  have  developed  a  shared 
situation  awareness  (Schaefer  et  al.  2016).  This  leads  directly  into  the  benefits  of 
the  inclusion  of  a  mobile  application  or  user  display  to  communicate  vehicle  intent 
to  the  passenger  and  adequate  warnings  in  situations  that  may  require  human 
intervention. 

5.3.4  Mobile  Application  or  User  Display  Design 

All  respondents  were  asked  if  a  cell  phone  application  or  onboard  user  display 
would  be  beneficial  to  provide  vehicle  information  to  riders.  Out  of  the  26 
respondents,  16  said  yes,  6  said  no,  and  4  did  not  respond.  In  addition,  7  respondents 
provided  suggestions  for  the  design  guidelines  relating  to  navigation-specific 
feedback,  error  reporting,  and  contact  information.  These  suggestions  and 
comments  ranged  from  simple  things  such  as  posting  hours  of  operation  and  contact 
information,  to  having  the  vehicle  provide  specific  route  information  and 
verification.  Table  4  provides  specific  feedback  and  comments  that  could  be 
applied  to  either  a  cell  phone  application  or  onboard  user  display. 

Table  4  Design  guidelines  for  a  user  display 


Number  of 

comments 

Specific  feedback 

4 

Navigation-specific  feedback 

•  Estimated  time  of  arrival 

•  Notification  of  change  in  vehicle  behaviors  (e.g.,  preparation  to  stop, 
turning,  destination  arrival) 

•  Easy  to  read  route  (e.g.,  stop  locations;  similar  to  GPS  navigation  system) 

•  Length  of  time  until  the  vehicle  arrives  at  the  pickup  location 

3 

Error  reporting 

•  Interactive  display  for  reporting  issues,  errors,  and  rider  concerns 

•  Denote  number  of  accidents  to  date  or  days  without  errors 

•  Provide  safety-related  information  and  comments 

1 

Contact  information 

•  Provide  contact  information  (e.g.,  phone  numbers  and  hours  of  operation) 
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Interestingly,  top  design  considerations  to  advance  passenger  trust  in  the  vehicle 
had  less  to  do  with  vehicle  autonomy  than  they  did  with  vehicle  functional 
performance.  In  other  words,  being  notified  of  information  such  as  estimated  time 
of  arrival  and  route  information  is  similar  to  information  one  might  either  ask  a 
human  driver  or  be  expected  to  have  available  in  a  vehicle  driven  by  a  human.  Even 
a  relatively  simple  suggestion  such  as  posting  contact  information  speaks  to  trust 
development  in  the  autonomous  system.  It  suggests  passengers  want  a  human 
fallback  option  to  call  in  case  of  error  or  if  they  become  confused  and  need  some 
aspect  of  the  system  explained  to  them,  especially  if  an  error  is  reported  and  the 
passenger  has  no  idea  what  to  do  next.  Teasing  out  the  reasons  behind  this  apparent 
desire  for  human  contact  is  an  area  of  future  research.  Additionally,  it  will  be 
important  to  understand  the  point  at  which  trust  in  the  system  supplants  this  need. 

6.  General  Discussion 


When  a  passenger  gets  into  a  vehicle  driven  by  a  human,  it  is  easy  for  them  to  put 
themselves  in  the  driver’s  place.  This  is  a  familiar  interaction  for  most  passengers, 
and  therefore  one  that  contributes  to  their  perceptions  of  safety  as  a  passenger.  It  is 
assumed  that  the  driver  is  using  sight,  sound,  and  experience  to  pay  the  right  amount 
of  attention  to  the  road  and  behaving  appropriately.  Similarly,  when  a  driver 
transitions  into  a  role  of  a  safety  operator,  it  is  also  assumed  that  there  is  an  extra 
level  of  safety  in  place  since  the  operator  is  ultimately  responsible  for  the 
passengers  and  bystanders.  When  the  vehicle  is  controlling  itself,  there  is  no  similar 
connection.  Passengers  have  only  their  general  preconceptions  of  autonomous 
systems  to  assess  the  appropriateness  of  decisions.  At  best,  the  average  passenger 
has  only  a  limited  understanding  of  the  sensor  systems  the  vehicle  is  using  to 
perceive  and  understand  its  environment,  until  they  have  ample  opportunity  to 
interact  with  the  system.  Most  of  the  questionnaire  responses  in  our  study  point  to 
the  fact  that  observation  and  interaction  with  autonomous  vehicles  can  increase 
feelings  of  trustworthiness  and  perceived  understanding  of  the  actual  autonomous 
capabilities  of  the  vehicle.  However,  for  those  with  a  limited  understanding  of  the 
autonomy,  there  is  a  need  to  move  beyond  just  advancing  the  engineering 
capabilities.  To  facilitate  interaction  and  an  efficient  transportation  system, 
passengers  and  bystanders  need  to  appropriately  trust  in  automated  driverless 
vehicle’s  ability  to  operate  safely  and  abide  by  norms  and  generally  accepted  rules 
of  the  road.  To  do  this  they  need  to  be  able  to  understand  how  the  vehicle  operates 
or  how  to  gather  information  about  its  operation  from  before  or  during  interaction. 
In  addition,  open-ended  feedback  from  respondents  suggests  the  need  for  a 
human-in-the-loop,  especially  when  the  safety  operator  is  removed  from  the 
vehicle. 
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The  general  public,  represented  in  this  study  by  a  random  sample  of  Soldiers  and 
civilians  in  and  around  the  WTB  complex,  has  very  little  direct  experience  with 
autonomous  technologies.  Expectations  and  perceptions  after  direct  experience 
affected  preconceptions  in  a  generally  positive  way.  Vehicle  design  should  include 
intuitive,  universal  interfaces  focused  on  providing  information  passengers  and 
pedestrians  could  use  to  make  a  connection  to  the  vehicle.  This  connection  and 
demonstrated  reliability  will  improve  trust  and  overall  performance  in  mixed  traffic 
environments. 

6.1  Key  Findings 

There  were  7  key  findings  from  this  work. 

1)  Most  people  had  positive  trust-related  ratings  despite  whether  or  not  they 
have  had  previous  direct  interaction  with  the  vehicle’s  autonomy. 

2)  People  believed  that  the  autonomous  capabilities  of  the  ARIBO  were 
acceptable  or  would  be  in  the  future,  regardless  of  whether  or  not  they  had 
seen  the  vehicle  operate  autonomously  or  not. 

3)  The  small  group  of  individuals  with  lower  perceptions  of  trustworthiness 
and  acceptability  were  found  to  have  a  minimal  understanding  the  system, 
suggesting  that  more  detail  or  information  prior  to  or  during  the  ride  could 
increase  feelings  of  safety  and  security. 

4)  Recommended  design  features  to  improve  passenger  comfort  were  specific 
to  the  potential  needs  of  the  wounded  Soldier  population  who  may  use  this 
vehicle  and  the  general  safety  of  all  riders.  These  included  more 
comfortable  seats,  extended  leg  room,  safety  belts,  and  effective 
communication  of  safety  features  and  options  to  all  passengers. 

5)  Recommended  design  features  to  increase  pedestrian  comfort  were  specific 
to  pedestrian  safety  awareness.  While  the  current  vehicle  capabilities  only 
had  the  vehicle  slow  down,  or  stop  and  wait  for  pedestrians,  feedback 
suggests  that  additional  alerts  or  warnings  (e.g.,  verbal  acknowledgment 
when  vehicle  is  paused  for  pedestrian,  hom  sounding  to  let  pedestrian  know 
the  vehicle  is  waiting,  or  signage  explaining  that  this  is  a  driverless  vehicle) 
would  improve  vehicle-pedestrian  interaction. 

6)  Recommended  design  features  to  increase  passenger  trust  in  the  vehicle 
were  specific  to  perceptions  of  safety,  specifically  error  prevention  and 
mitigation.  Specific  features,  such  as  vehicle  size,  coloration,  or  signage, 
can  communicate  to  other  road  users  that  the  vehicle  is  a  driverless  vehicle, 
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thus  impacting  vehicle-to-vehicle  interaction.  In  addition,  it  is  essential  to 
identify  methods  for  determining  when  and  how  a  passenger  can  adapt 
control  allocation  and  respond  to  high-risk  situations  once  the  operator  is 
removed  from  the  vehicle. 

7)  The  system  would  benefit  from  a  user  display  (e.g.,  mobile  application  or 
onboard  display)  to  communicate  vehicle  awareness  and  actions,  including 
but  not  limited  to  navigation-specific  feedback  (e.g.,  estimated  time  of 
arrival,  changes  in  behaviors,  routes,  arrival  time),  error  reporting  (e.g., 
means  to  report  issues  or  errors,  number  of  errors  or  accidents, 
safety-related  information),  and  a  means  to  connect  to  a  human  monitor. 

6.2  Limitations 

There  were  2  potential  limitations  to  this  work. 

1)  Limited  Ridership:  Despite  the  fact  that  WTB  averages  more  than  700 
appointments  at  WAMC  each  week  and  that  the  barracks  are  capable  of 
housing  more  than  600  Soldiers  at  a  given  time,  there  was  limited  response 
from  the  WTB  Soldiers.  This  was  in  part  impacted  by  a  limited  number  of 
scheduled  riders  using  the  ARIBO  Mobile  application,  including  12 
scheduled  riders  with  112  rides  during  Phase  1  of  operation  and  5  scheduled 
riders  with  a  total  of  15  rides  during  Phase  2  of  operation.  In  addition  to  the 
above  riders,  unscheduled  riders  and  Soldiers  who  responded  to  email 
requests  for  feedback  made  up  most  of  the  respondents.  However,  specifics 
as  to  the  amount  of  total  riders  and  rides  were  not  recorded.  These 
limitations  in  sample  size  limited  the  type  of  statistical  analysis  that  could 
be  conducted  and  directly  influenced  the  potential  outcomes  of  this  work. 

2)  Trust:  Subjective  assessment  of  trust  in  an  autonomous  vehicle  is  not 
restricted  to  the  engineering  technologies  of  the  vehicle  alone.  Trust  is 
assessed  with  respect  to  a  whole  system.  For  the  ARIBO  vehicle  this  system 
included  a  driver  or  safety  operator,  the  users  or  potential  users  at  the  WTB 
complex,  and  the  direct  support  of  the  command  staff  for  vehicle  integration 
and  use.  Therefore,  it  is  possible  that  the  respondents  assumed  that  the  larger 
Army  would  not  allow  a  technology  that  was  not  safe  or  function 
appropriately  to  be  used  by  the  WTB  Soldiers.  Therefore,  based  on  the 
current  questionnaire  design,  it  is  not  possible  to  vet  the  trust-related 
responses  without  additional  follow-up  research. 
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7.  Conclusions 


Although  this  user  perception  study  had  far  less  participation  than  anticipated,  a 
number  of  areas  for  future  research  were  highlighted  that  are  relevant  to  robotic 
vehicle  deployment  in  the  Army.  In  general,  individuals  have  limited  or  no 
experience  with  robotic  technologies,  such  as  driverless  vehicles,  but  minimal 
exposure  to  these  technologies  can  positively  impact  trust  ratings,  as  found  in  the 
study.  This  finding  could  be  specific  to  the  vehicle  capabilities  and  design  for  the 
use  of  personnel  transport,  but  could  also  be  influenced  by  the  larger  operations 
space.  For  example,  trust  in  the  vehicle  could  be  directly  influenced  by  direct  access 
to  the  driver  or  safety  operator.  How  this  trust  changes  when  a  safety  operator  is 
removed  from  the  vehicle  is  a  future  line  of  research.  These  findings  may  have  also 
been  influenced  by  reputation  of  the  larger  Army  such  that  individuals  may  feel 
that  the  Army  would  not  allow  nor  encourage  ridership  if  the  design  and  equipment 
were  not  safe — this  is  a  thought  that  will  have  to  be  tested  with  data  collected  in  the 
future. 

While  the  ARIBO  shuttle  vehicles  were  rated  positively  across  the  trust  and 
acceptability  questions,  specific  design  guidelines  were  provided  to  improve  trust 
and  comfort  by  passengers,  pedestrians,  and  other  road  users.  Most  of  the  feedback 
supported  the  need  for  increased  communication  either  prior  to  use  or  during 
operations.  Increased  communication  for  passengers  and  pedestrians  would  include 
education  on  the  vehicles  technology  and  capabilities.  Increased  communication 
for  vehicle-to-vehicle  interactions  would  include  ways  of  letting  other  vehicles 
know  that  the  vehicle  had  detected  the  other  vehicle  and  its  plan  of  action.  Overall, 
these  findings  support  the  advancement  of  robotic  technology  and  start  to  show  that 
such  technology  may  provide  major  advantages  for  the  Army  in  the  area  of 
manned-unmanned  teaming. 
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Appendix  A.  Previous  Simulation  Study  Findings 
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Previous  findings  recommended  4  areas  of  design  recommendations  to  increase 
situation  awareness  to  engender  trust  in  the  Applied  Robotics  for  Installations  and 
Base  Operation  (ARIBO)  vehicle.1 


1.  External  Environmental  Awareness:  Passenger  wants  to  know  that 
the  vehicle  knows  what  is  going  on  in  the  environment 

•  13  out  of  20  participants  provided  14  design  recommendations 

•  Examples  include  obstacle  detection  indicators  on  an  onboard 
user  display,  response  time  indicator  for  vehicle  behavior,  and 
visual/auditory  pedestrian  safety  warnings  and  alerts. 

2.  Navigation-based  Autonomy:  Passenger  wants  to  know  where  the 
vehicle  is  currently  located  and  where  it  is  going 

•  15  out  of  20  participants  provided  26  recommendations 

•  Examples  include:  pickup  and  drop-off  locations  on  an  onboard 
or  mobile  display,  estimated  time  of  arrival  indicator,  and 
visual  indicators  such  as  turn  signals  to  communicate  changes 
in  vehicle  direction 

3.  System  Awareness:  Passenger  wants  to  know  that  the  vehicle  is  aware 
of  its  current  state 

•  9  out  of  20  participants  provided  1 1  recommendations 

•  Examples  include  activation  lights,  speed  indicator,  display 
that  shows  changes  in  vehicle  behaviors,  vehicle  health  status 
such  as  fuel  or  battery  life,  and  countdown  until  a  specific 
vehicle  action 

4.  Passenger  Awareness  and  Safety:  Passenger  wants  to  make  sure  the 
vehicle  is  aware  of  those  onboard 

•  6  participants  provided  7  recommendations 

•  Examples  include  auditory  or  visual  safe  boarding  or  exiting 
alerts,  notice  of  engagement  of  safety  features,  weather 
indicators 


1  Schaefer  KE,  Brewer  R,  Putney  J,  Mottern  E,  Barghout  J,  Straub  ER.  Relinquishing  manual 
control:  collaboration  requires  the  capability  to  understand  robot  intent.  Proceedings  of  the  9th 
International  Workshop  on  Collaborative  Robotics  and  Human  Robot  Interaction;  2016  Oct  31-Nov 
4;  Orlando,  FL.  2016.  Los  Alamitos  (CA):  IEEE  Computer  Society;  c2016.  p.  359-366. 
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Appendix  B.  Questionnaires 
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ARIBO  Driverless  Vehicle:  Phase  1  Questionnaire 

1 .  Circle  the  most  appropriate  description  of  your  interaction  with  the  vehicle: 

Driver  Passenger  Observer  (not  riding  on  vehicle) 

2.  How  many  times  have  you  been  a  passenger  on  this  vehicle? _ 

3.  Is  this  your  first  time  filling  out  this  questionnaire  (circle  one)? 

Yes  No 

4.  Have  you  seen  the  vehicle  operate  autonomously  without  a  driver  controlling  it  (circle 
one)? 

Yes  No 


Directions:  Please  rate  your  agreement  with  each  item  based  on  your  beliefs  about  the 
FUTURE  capabilities  with  the  vehicle. 

5.  The  vehicle  will  be  intelligent  (circle  one) 


Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 

6.  The  vehicle  will  control  itself  (circle  one) 


Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 

7.  The  vehicle  will  be  used  regularly  by  Soldiers  at  Ft.  Bragg  (circle  one) 


Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 

,  The 

vehicle  will  be 

safe  (circle  one) 

Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 

.  The 

vehicle  will  be 

trustworthy  (circle  one) 

Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 
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Directions:  Please  rate  acceptability  of  each  item  based  on  your  beliefs  about  the  FUTURE 
capabilities  of  the  vehicle. 

10.  The  vehicle  will  be  able  to  navigate  throughout  the  medical  facilities  without  human 
intervention. 

Totally  Unacceptable  Slightly  Neutral  Slightly  Acceptable  Perfectly 

Unacceptable  Unacceptable  Acceptable  Acceptable 

1 1 .  The  vehicle  will  be  able  to  avoid  other  vehicles,  obstacles,  and  pedestrians  without 
human  intervention. 

Totally  Unacceptable  Slightly  Neutral  Slightly  Acceptable  Perfectly 

Unacceptable  Unacceptable  Acceptable  Acceptable 

12.  The  vehicle  will  be  able  to  respond  to  traffic  rules  (e.g.,  road  signs,  road  rules)  without 
human  intervention. 

Totally  Unacceptable  Slightly  Neutral  Slightly  Acceptable  Perfectly 

Unacceptable  Unacceptable  Acceptable  Acceptable 

Directions:  The  following  questions  are  open-ended  questions  that  will  allow  you  provide 
recommendations  specific  to  the  future  design  of  the  vehicle. 

13.  Please  provide  any  recommendations  for  the  design  of  the  vehicle  in  order  to  increase  the 
comfort  of  a  passenger  onboard. 


14.  Please  provide  any  recommendations  for  the  design  of  the  vehicle  in  order  to  increase  the 
comfort  of  nearby  pedestrians. 


15.  Please  provide  any  recommendations  for  the  design  of  the  vehicle  that  would  increase 
trust  in  the  vehicle. 


16.  Should  we  provide  passengers  with  a  cell  phone  app  or  onboard  display  that  provides 
vehicle  information  (circle  one)? 

Yes  No 
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ARIBO  Driverless  Vehicle:  Phase  2  Questionnaire 

1 .  Circle  the  most  appropriate  description  of  your  interaction  with  the  vehicle: 

Safety  Operator  Passenger  Observer  (not  riding  on  vehicle)  Other 

2.  How  many  times  have  you  been  a  passenger  on  this  vehicle? _ 

3.  Is  this  your  first  time  filling  out  this  questionnaire  (circle  one)? 

Yes  No 

4.  Have  you  seen  the  vehicle  operate  autonomously  without  a  driver  controlling  it  (circle 
one)? 

Yes  No 


Directions:  Please  rate  your  agreement  with  each  item  based  on  your  PAST/CURRENT 
experiences  with  the  vehicle. 

5.  The  vehicle  is  intelligent  (circle  one) 


Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 

6.  The  vehicle  controls  itself  (circle  one) 


Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 

7.  The  vehicle  will  be  used  regularly  by  Soldiers  at  Ft.  Bragg  (circle  one) 


Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 

8.  The  vehicle  is  safe  (circle  one) 


Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 

9.  The  vehicle  is  trustworthy  (circle  one) 


Strongly 

Disagree 

Somewhat 

Neither 

Somewhat 

Agree 

Strongly 

Disagree 

Disagree 

Agree  nor 
Disagree 

Agree 

Agree 

Directions:  Please  rate  acceptability  of  each  item  based  on  your  PAST/CURRENT  experiences 
with  the  vehicle. 

10.  The  vehicle  navigates  throughout  the  medical  facilities  without  human  intervention. 


Totally  Unacceptable  Slightly  Neutral  Slightly  Acceptable  Perfectly 

Unacceptable  Unacceptable  Acceptable  Acceptable 
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1 1 .  The  vehicle  avoids  other  vehicles,  obstacles,  and  pedestrians  without  human  intervention. 


Totally  Unacceptable  Slightly  Neutral  Slightly  Acceptable  Perfectly 

Unacceptable  Unacceptable  Acceptable  Acceptable 

12.  The  vehicle  responds  to  traffic  rules  (e.g.,  road  signs,  road  rules)  without  human 
intervention. 

Totally  Unacceptable  Slightly  Neutral  Slightly  Acceptable  Perfectly 

Unacceptable  Unacceptable  Acceptable  Acceptable 


Directions:  The  following  questions  are  open-ended  questions  that  will  allow  you  provide 
recommendations  specific  to  the  future  design  of  the  vehicle. 

13.  Please  provide  any  recommendations  for  the  design  of  the  vehicle  in  order  to  increase  the 
comfort  of  a  passenger  onboard. 

14.  Please  provide  any  recommendations  for  the  design  of  the  vehicle  in  order  to  increase  the 
comfort  of  nearby  pedestrians. 

15.  Please  provide  any  recommendations  for  the  design  of  the  vehicle  that  would  increase 
trust  in  the  vehicle. 


16.  Should  we  provide  passengers  with  a  cell  phone  app  or  onboard  display  that  provides 
vehicle  information  (circle  one)? 


Yes  No 


If  so,  what  type  of  information  should  be  provided  to  help  increase  a  passenger’s  trust  in 
the  vehicle? 
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Intentionally  Left  Blank. 


Approved  for  public  release;  distribution  is  unlimited. 


34 


List  of  Symbols,  Abbreviations,  and  Acronyms 


ARIBO 

Applied  Robotics  for  Installations  and  Base  Operations 

GPS 

global  positioning  system 

ODOA 

obstacle  detection  and  obstacle  avoidance 

TARDEC 

US  Army  Tank  Automotive  Research,  Development  and 
Engineering  Center 

WAMC 

Womack  Army  Medical  Center 

WTB 

Warrior  Transition  Battalion 
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